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The Education Endowment Foundation 


The Education Endowment Foundation (EEF) is an independent grant-making charity dedicated to 
breaking the link between family income and educational achievement, ensuring that children from all 
backgrounds can fulfil their potential and make the most of their talents. 


We aim to raise the attainment of children facing disadvantage by: 


e Identifying promising educational innovations that address the needs of disadvantaged 
children in primary and secondary schools in England; 


e Evaluating these innovations to extend and secure the evidence on what works and can be 
made to work at scale; 


e Encouraging schools, government, charities, and others to apply evidence and adopt 
innovations found to be effective. 


The EEF was founded in 2011 by lead charity The Sutton Trust, in partnership with Impetus Trust 
(now part of Impetus — The Private Equity Foundation), with a £125m grant from the Department for 
Education. With investment and fundraising income, the EEF intends to award as much as £200m by 
2026. Together, the EEF and Sutton Trust are the Government-designated What Works Centre for 
Improving Education Outcomes for School-Aged Children. 
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Literacy Catch-up 


In May 2012 the Department for Education awarded the EEF a further £10 million for a grants round 
dedicated to literacy catch-up projects for children at the primary-secondary transition. The projects 
funded within this round aimed to identify effective ways to support pupils who do not achieve Level 4 
in English by the end of Key Stage 2. 
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About the evaluator 


The project was independently evaluated by a team from Durham University led by Professor Stephen 
Gorard. 


Stephen Gorard is Professor of Education and Well-being, and Fellow of the Wolfson Research 
Institute at Durham University. He is a Methods Expert for the US government Institute of Education 
Science, member of the ESRC Grants awarding Panel, and Academician of the Academy of Social 
Sciences. 


His work concerns the robust evaluation of education as a lifelong process, focused on issues of 
equity and effectiveness. He regularly gives advice to governments and other policy-makers, including 
oral and written advice to the House of Commons Education Committee every year since 2003. He is 
also a widely read and cited methodologist, involved in international and regional capacity-building 
activities, and used regularly as an adviser on the design of evaluations by central and local 
governments, NGOs and charities. He is currently an evaluator for the European Commission 
Directorate-General for Education and Culture, the Department of Work and Pensions, the Food 
Standards Agency, the Learning and Skills Information Service, and the Education Endowment 
Foundation. He is author of nearly 1,000 books and papers. 


Contact details: 


Professor Stephen Gorard 
School of Education 
Durham University 
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Executive Summary 


Executive Summary 


The project 


The Future Foundations Society CIC (Future Foundations) summer school programme is a literacy 
and numeracy catch-up intervention which provided extra schooling in the summer holidays. Pupils 
attending the four-week programme followed a specially designed curriculum involving regular literacy 
and numeracy lessons taught by trained primary and secondary school teachers. Lessons were 
supported by mentors and peer-mentors and generally conducted in small teaching groups. Each 
afternoon, pupils participated in a variety of sports and enrichment activities. The programme took 
place across three sites in London and the South East: Brighton, Enfield and Islington in the summer 
of 2013. It was targeted at pupils in Years 5 and 6 who were eligible for free school meals (FSM) and 
at pupils who had not achieved Level 4 in English or maths at the end of Key Stage (KS) 2. 


In 2012, Future Foundations developed, organised and piloted the summer school on one site, in 
Enfield. Recommendations from the formative evaluation of this pilot, also funded by the Education 
Endowment Foundation informed the development of the 2013 summer school. 


What impact did it have? 


The overall result on English outcomes was an effect size of +0.17. This effect can be envisaged as 
suggesting that on average pupils receiving the intervention would make approximately two additional 
months’ progress over the course of a year compared to similar pupils who did not. This is similar to 
pupils’ normal rate of progress in term time. 


The overall result on maths outcomes was an effect size of 0. This effect can be envisaged as 
suggesting that on average pupils receiving the intervention would make no additional progress over 
the course of a year compared to similar pupils who did not. 


The evaluation identified different results for specific groups of pupils, though conclusions about 
groups of pupils are necessarily more tentative than the overall finding. Positive benefits were 
suggested in English for pupils eligible for FSM, who made two additional months’ progress on 
average, and for pupils in Year 5. In English, boys also appeared to benefit from the programme more 
than girls, making three additional months’ progress compared to one additional months’ progress for 
girls. 


However, the intervention also appeared to have a negative impact on pupils eligible for FSM in 
maths. Understanding this negative impact is challenging, but it is possible that teaching of maths was 
of a lower quality than of English, leading to poor behaviour and disengagement, or that English 
requires more continuous work whereas maths is more resistant to summer learning loss. 


The programme was implemented successfully on all three sites. Despite considerable efforts from the 
developers, a significantly smaller number of pupils attended the school than had been hoped for, with 
less than half of the target number of pupils signing up for the programme. A number of pupils also 
dropped-out once the programme had started. These challenges were particularly apparent in the 
numbers recruited outside of London (only 19% of target in Brighton compared to 34% of target in 
Islington and 44% in Edmonton), suggesting that there may be problems were the programme to be 
rolled out, particularly to less densely populated areas. 


It is clear that many pupils enjoyed their time at the summer school, and it is possible that the 
programme led to non-academic benefits for the pupils and their families. The programme was 
popular with parents, who appreciated the free provision of academic and enrichment activities over 
the summer holidays. 
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Executive Summary 


Overall English 


Overall maths 


FSM-eligible English 


FSM-eligible maths 


Boys only English 


Girls only English 


Boys only maths 


Girls only maths 


*Evidence ratings are a new measure under development based on a number of factors including study type, size and drop-out. 
Ratings are provisional and are not given for sub-group analyses, which will always be less secure than overall findings. 


For more information about ratings visit: www.educationendowmentfoundation.org.uk/evaluation. 


How secure is this finding? 


The evaluation was set up as a small-scale efficacy trial to test the impact of the summer school. 
Efficacy trials seek to test evaluations in the best possible conditions to see if they hold promise, but 
do not seek to demonstrate that the findings hold at scale in all settings. To test this question, a future 
evaluation run on a larger scale in a wider variety of areas could be conducted. 


The findings are based on a randomised controlled trial, with individual random allocation to groups to 
the summer school or a control group who did not receive the intervention. The study was designed to 
involve 1,000 pupils. However, the problems with attracting pupils and keeping them in the project 
meant that the results are substantially weakened. The actual population of pupils who fitted the 
criteria set out for the programme caused an issue. After this group had been exhausted recruitment 
took place across a wider range of schools. 


Ultimately, 435 pupils signed up to the trial, which decreased the power of the study, while the high 
levels of dropout increased the risk that the findings are biased, as the pupils who attended the 
summer school may have been systematically different from those who did not. As a result, the 
findings cannot be taken as definitive. This study suggests that it is challenging to assess summer 
schools using randomised designs, as many families are unwilling to wait to know whether their child 
will be selected to attend. 


Though caution is essential, there is some promise from the results of progress in English that might 
be worth pursuing in the future. The headline finding for English has been confirmed by a number of 
alternate analyses including regression modelling and post-test only, which strengthens the case for 
further work in this area. This short-term evaluation does not assess the long-term impact of attending 
the summer school, though long-term outcomes will be measured through the National Pupil Database 
in the future. 


The pre-existing evidence on the impact of summer schools is very mixed. A number of studies have 
been conducted, most commonly in the US, but these have often been methodologically weak, 
meaning that further study in this area is required. 


Education Endowment Foundation 6 


Executive Summary 


How much does it cost? 


The cost of the approach is estimated at £1,370 per pupil. This estimate includes administration, 
resources and activities (estimated at £350 per pupil), salary costs and training (£835) and food and 
transport (£185). Estimates are based on 160 pupils attending a school on a single site. 


1. Attracting pupils to the summer school, and maintaining high attendance throughout the programme, was challenging. 


2. As aresult of the trial’s eventual size and the level of pupil dropout, the overall findings of the programme on English or 
maths are not definitive. 


3. However, there is evidence of promise for English, particularly for FSM-eligible and Year 5 pupils, which may warrant 
further study. 


4. The programme was relatively expensive. As a way of improving academic outcomes alternative approaches delivered 
during the school year may provide similar benefits for a lower cost. 


5. Future evaluations could explore whether apparent gains for progress in English continue into the secondary phase. 
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Introduction 


Intervention 


This report documents the results of an evaluation of the Future Foundations academic summer 
school which took place in August 2013. The programme ran daily for four weeks and provided pupils 
with literacy and numeracy support in the morning and enrichment activities in the afternoon. 


There is some evidence of a summer learning loss in the UK, especially for children as they move 
from primary to secondary school, and one proposal to address this is the provision of summer holiday 
schools for economically disadvantaged pupils who are reaching the end of their primary schooling. 
The programme was intended to improve the outcomes of Year 5 and 6 pupils from disadvantaged 
backgrounds by offering small group academic tuition and the opportunity to take part in a wide range 
of enrichment activities. 


The academic content of the intervention focused on attainment in literacy and numeracy. The 
programme is loosely based on the US BELL Summer School and BELL Accelerated Learning 
programmes, and is built on a pilot programme conducted by Future Foundations in London in 2012. 


All of these interested parties have evaluated their own work as a success, and there are indeed many 
indicators of success in terms of satisfaction and attitudinal measures. However, none has yet 
convincingly demonstrated a beneficial impact on pupil learning for the Year 5 and 6 age groups, as 
assessed by formal testing with a true counterfactual. 


Background and existing evidence 


In September 2011, the EEF announced its intention to fund a pilot of a US-style academic summer 
school in England, to assess the feasibility of summer school programmes relevant to literacy and 
numeracy learning outcomes. In April 2012, after an open competition run by the EEF with support 
from Building Educated Leaders for Life (BELL), a major US summer school provider, it was 
announced that Future Foundations would run the pilot in Nightingale Academy, Edmonton with the 
Edmonton Schools Partnership. Funding was provided for a four-week summer school for 160 pupils 
reaching the end of Year 5 or 6. The programme was intended to target pupils from disadvantaged 
backgrounds, who were underperforming at their expected levels, and likely to benefit from 
participation in the programme. One of the reasons for the pilot programme is that there is little robust 
evidence on the efficacy of the BELL approach, especially in the UK. The situation was not then at the 
point of equipoise (where there is indicative evidence only) needed as the basis for a full trial. 


The existing evidence for the efficacy of summer schools from the US is not yet substantial (see 
below). The pilot was therefore designed to test the feasibility of organising a summer school in a 
relatively deprived area. In particular, it sought to assess: whether there was demand for the 
programme, whether families would support and sustain the programme, and whether professional 
staff would be willing to work during their summer holidays. The pilot also provided an opportunity to 
test the training, curriculum, activities, delivery and management of the programme, so that the 
programme could be improved if the pilot was determined to have demonstrated its feasibility. On the 
condition that feasibility was demonstrated, the EEF announced its intention to run a larger trial of the 
summer school in 2013, to establish its impact on attainment. 
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The evaluation was funded as part of a £10m literacy catch-up round funded by the Education 
Endowment Foundation. Schools and other educational organisations were encouraged to develop 
programmes that could support disadvantaged pupils to catch up. The catch-up literacy projects are a 
set of educational interventions for pupils who are struggling to reach the age appropriate levels. The 
round, announced by the Deputy Prime Minister in May 2012 was motivated by existing evidence that 
struggling pupils entering secondary school are more likely to remain behind across the curriculum 
compared to classmates (https://www.gov.uk/government/news/10-million-to-boost-literacy-for-year- 
sevens). 


US evidence on summer schools 


Previously, Chaplin and Capizzano (2006) reported what was intended to be a randomised evaluation 
of the BELL summer programme in Boston and New York. A total of 1,917 pupils applied to the 
programme, of which only 1,225 agreed to be part of the study, yet the random allocation to treatment 
or control was of the original 1,917. The report is not very clear on the numbers finally appearing in the 
treatment and control groups. Of the 1,225 consenting, 138 were excluded from the study, leaving 
1,087. But in the paper, data are only presented for 835 cases (44% of the original applicants). This 
means that the study can no longer be regarded as randomised in nature. The summer programme 
involved both maths and reading, but the results are only presented for reading. The difference in test 
scores between treatment and control groups were negligible, and provide no solid evidence of a 
beneficial impact. The report claims that this is because the control group had 16 more days in school 
before the post-test than the treatment group did. This design flaw may indeed be the reason for the 
lack of a more positive result, but the study does not demonstrate a positive result. The report uses 
statistical significance, incorrectly in this context of a non-random sample. 


The overall effect size of this summer programme on reading, not calculated in the report, was only 
around +0.02. And the performance of the control group in vocabulary was actually better than the 
treatment. This report therefore provides very far from convincing evidence of effect, and it was 
generated by authors with a potential conflict of interest. 


Harvard Family Research Project (2006), clearly with a link to BELL and so a potential conflict of 
interest, presented a number of evaluations of the BELL summer schools and accelerated learning 
programme. These include BELL (2001, 2002 and 2003). The 2002 programme took place in 
Washington DC, Baltimore, Boston and New York, and was intended to raise reading, writing and 
maths scores for children from 1st to 7th grade. Further sites were added in 2006. Programmes were 
either four or six weeks in duration. The reports are unclear which sites were included in each 
analysis. No improvement is reported for writing. A ‘significant’ gain is reported for reading but not 
maths following the four-week programme, but the effect size (not reported) is small. The gains were 
higher in both maths and reading for the six-week programme, although low-income children had 
lower gains and in several years actually lost ground. Higher gains were generally reported for the 
younger year groups. 


British evidence on summer schools 


In September 2011, the Deputy Prime Minister announced that £50 million would be made available 
for a summer schools programme in England. The scheme was intended to support disadvantaged 
pupils in the transition phase from primary to secondary school. Over 1,700 schools conducted 
summer school programmes in 2012, and in the summer of 2013 over 1,900 schools conducted 
summer programmes sponsored by the Department for Education. An evaluation of summer school 
impact was conducted by NFER in 2013, in which nearly 21,000 pupils participated (Marting et al. 
2013). The study involved a large-scale survey in which the target group consisted of pupils in schools 
that conducted the summer school programme and pupils in comparator schools that did not 
participate. The report suggested a positive impact from summer school programmes on pupils’ 
confidence, social skills and readiness to attend their secondary school, rather than on attainment. 
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See et al. (2012) conducted a rapid review of the evidence on the transition from Year 6 to Year 7 in 
England, and the possibility of pupils catching up in literacy. They came across two further studies 
concerning the impact of summer school programmes. One study of around 2,000 pupils in transition 
from primary to secondary divided them non-randomly into two groups. It found no differential impact 
on literacy gain scores between the groups who attended a 50-hour summer literacy school compared 
to a control. Both groups demonstrated an equivalent decline in scores from pre- to post-test 
(Sainsbury et al. 1998). Therefore, it seems that the reason for any decline over that crucial summer 
period is not to do with whether literacy practice and teaching takes place. It could be due to anxiety 
about changing school, a change in school routine or a different curriculum emphasis. A smaller, more 
recent study from the US involved 331 pupils from Grades 1 to 5 in one school (Kim 2006). Using 
stratification in terms of pre-test reading ability, pupils were randomly allocated to a treatment or 
delayed treatment in a waiting-list design. The treatment involved receiving 10 free books to read 
during the summer vacation, including postcards and letters to stimulate reading. Using self-report, the 
treatment group read three more books, on average, than the control. However, this did not convert to 
any difference in the literacy scores between the groups after the vacation. The number of pupils is 
quite small in the preferred age range (grade 5) and 52 pupils moved away during the summer 
(proportionately for each group and stratum). Put another way, what these two studies may show is 
not that summer interventions cannot work, but that it is not just about doing something well-meaning 
and plausible in summer. For example, it may be necessary to have some further input rather than just 
providing books. On the other hand, the first study suggests that simply having more ‘school’ over 
summer does not help either. As far as we can tell from the evidence here, summer school 
programmes have not yet been shown to be effective in improving literacy for pupils in transition. 


Overall then, there is mixed but not very promising existing evidence of a beneficial impact from such 
a summer programme for older primary school pupils lasting only four weeks. What is needed is much 
firmer evidence. 


Evaluation objectives 


. To evaluate the impact of a four-week summer school intervention on the literacy and 
numeracy performance of Year 5 and 6 volunteer pupils 
. To contextualise this impact in terms of the background characteristics of the pupils 


Project team 


The programme was developed and conducted by Future Foundations, an independent training 
organisation which designs and delivers diverse academic and enrichment activities for schools. The 
summer school 2013 included a team of trained teachers who were interviewed and selected by 
Future Foundations. There were also mentors and peer-mentors who provided extra support to the 
teachers. 
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Methods 


Trial design 


The intervention was proposed and designed as a simple individually randomised controlled trial of 
1,000 pupils divided into two equal groups — a treatment group and a control without placebo or 
waiting list. The first group received the treatment over summer 2013. The prior attainment scores 
consisted of KS2 fine point scores from summer term 2013. These were obtained from schools via the 
developer and represent each child’s summer test scores for reading, writing and maths. Post-tests 
(GL Assessment’s Progress in English and Progress in Maths) were administered in groups in the 
eventual schools attended by both groups of pupils in autumn 2013. Primary schools were generally 
cooperative in conducting the pencil-and-paper post-tests delivered to each of the schools by the GL 
assessment team. It was harder to get agreement from secondary schools to test the original Year 6 
pupils after they had begun Year 7. 


A great deal of effort was put in by the developers to reduce demoralisation and consequent dropout. 
This involved not revealing the groups until after the randomisation, use of a refundable deposit for 
registrants, and neutral administration of the post-test. 


Eligibility 


Pupils: The programme was targeted at pupils aged nine to 11 (Years 5 and 6) who were eligible for 
FSM, had low literacy scores, English as an additional language, or a special educational need. Pupils 
in Year 6, going into Year 7, also had to be due to attend the secondary school at which the summer 
programme was hosted. Pupil participation was voluntary because the intervention took place in the 
summer holiday period. No pupil was excluded from the programme after selection but some chose to 
drop out at a very early stage. The reasons for dropout are explained in the description of the sample 
size. 


Schools: Future Foundations was successful with responses from 43 schools in areas of relatively 
high deprivation in Brighton, Edmonton and Islington. These schools have a large pupil population 
from disadvantaged backgrounds, and the idea of a four-week free summer programme was received 
positively. Three local secondary academies, one in each of the three sites, collaborated with Future 
Foundations to provide the required premises and other facilities to conduct the programme. The 
school-level characteristics of all these schools are in the Appendix. All schools were co-educational. 


In collaboration with the participating schools, Future Foundations sent out an invitation to attend the 
programme to all relevant pupils, with a participation consent form for parents. The invitation letters 
were sent to parents by Future Foundations before the randomisation was conducted by the 
evaluation team. The letters clearly stated that eventual participation would be subject to 
randomisation results. An extensive programme of recruitment visits was organised to primary and 
secondary schools before the programme to generate interest amongst parents and pupils. 
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Intervention 


Over the four weeks, all pupils participated in two 75-minute academic lessons each morning, one for 
literacy and one for numeracy. These were delivered to small tuition groups of a planned 10+ pupils by 
high-quality teachers, supported by two mentors (one of which was a sixth-former or other pupil), 
using a scheme of work designed specifically for the programme. In the afternoons, pupils took part in 
a diverse set of enrichment activities, including sports, arts, cookery, off-site theatre and raft building. 
The programme closed with a special graduation ceremony for pupils and parents/carers. 


Academic sessions 


Each day included small group teaching in literacy and numeracy. The main aspects of the academic 
activities were: 


. Structured lesson plans (75 minutes for each class session for numeracy and literacy) 
with clearly written aims for every activity 
. A specific summer programme reading book 
. Individual pupils’ portfolios 
. Teacher evaluation and feedback 
Lesson plans 


The lesson plans were designed by experts in each area in preparation for the summer, using 
formative feedback from the pilot study. Teachers were given details on the aims of each lesson, and 
the activities and resources they were supposed to conduct in the sessions 


An example of each lesson was observed by one lead coordinator in all three sites. Their job was to 
give feedback to the teachers, and, by regularising the process of teacher observation and feedback, 
to standardise the intervention across the sites and maintain quality of teaching. This feedback model 
was similar to the Ofsted model for teacher evaluation. 


Pupils’ portfolio 


Pupils were given portfolios which were complete activity books for literacy and numeracy learning 
activities. These included the resources, worksheets, evaluations and progress reviews for every 
lesson of the four weeks. The portfolios were devised to allow gradual progress in performance, and 
demonstrate this to the children, their parents, and those evaluating the programme. Each portfolio 
was also a record of pupils’ learning. 


Teacher feedback to pupils 


All pupils were seen working on the portfolio and teachers/mentors were giving their feedback on 
performance. Other regular rewards included stars, stickers, and merit awards. 


Pupil feedback to teachers and mentors 


At the end of each lesson there was a pupil feedback corner to tell the teacher whether they had 
achieved their learning objectives, enjoyed the activity, or found difficulties. 
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Teacher transition in Week 3 


The teachers were generally recruited to work for only two of the four weeks. In the third week there 
was a teacher’s transition phase and a new batch of teachers took over. A handover session was 
organised on the Friday of Week 2 to enable all new teachers to meet their colleagues, learn about 
pupils and develop an understanding of the rhythms and routines on their site. The mentors and peer- 
mentors remained the same. According to Future Foundations this transition was necessary in order 
to get good teachers to work during the summer holidays and to ensure they would be able to give 
their full energy to the programme and return to school rested. Many working teachers could only be 
engaged for a two-week period, so in order to make it standard practice, all teachers (even if they 
could work additional weeks) changed in the third week. Teachers in both of the phases were given 
the same training and were informed about this change when they were first recruited. The portfolio 
was a record of each pupil performance so that the second batch of teachers could assess how each 
pupil had been performing previously. 


Progression Monitoring Sheet (PMS) 


The PMS was an addition to the programme to help ensure the smooth transition between teachers. 
This was completed by mentors working closely with pupils, and made available to the teachers. 
Feedback from mentors who were likely to be around for the whole duration of the programme was 
found useful by the new group of teachers starting in the third week. 


Enrichment activities 


The afternoon sessions were for a range of enrichment activities. Future Foundations hired external 
providers according to each site who could arrange a variety of activities over the four weeks. The 
activities included drama performance, art and creativity, science challenges, cooking, dance, music, 
and sports such as football, indoor cricket, fencing, and gym exercises. There were also arrangements 
for outdoor activities such as raft building, swimming and theatre performance. To some extent these 
activities varied by site, with London having a greater range of cultural and other resources within easy 
travel distance. 


Sample size 


The summer school was aimed at pupils leading up to the transition phase, so the sample included 
primary pupils from Years 5 and 6. Future Foundations had successfully recruited to the summer 
school 2012 in Edmonton. They aimed to increase the sample to 1,000 pupils for 2013, of which 500 
would be in each arm of the trial. 


In practice, the developers found recruitment much harder than anticipated, especially in the sites 
other than Edmonton. One of the original sites was abandoned due to the partner selected to host not 
being able to secure a school committed to the full trial. Islington was therefore introduced instead 
quite late in the process. The initial target of 1,000 pupils was overoptimistic and any future 
evaluations should more carefully consider the numbers of eligible pupils in a particular area prior to 
agreeing targets. 


It is not clear to what extent this is a problem for less populated areas (since recruitment was harder in 
Brighton than Edmonton, for example), lack of appeal of a summer programme, or concerns about 
being part of the evaluation. In more populous areas it was easier to recruit pupils, but other factors 
impacting on recruitment may have included concerns about being part of the evaluation, alternative 
summer activities already having been booked or the lack of appeal of a summer programme. 
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Early applicants clearly wanted to know whether they were part of the programme (so randomisation 
was needed early), but recruitment was running late, with less than a quarter of the target achieved by 
the initial early June 2013 deadline. The evaluators agreed to randomise in three waves so that each 
batch of applicants would know early whether they were free for summer or not. The evaluators also 
agreed with the developers and funders to have a larger treatment than control group to make the 
programmes viable. In addition, funders requested that more Year 6 pupils be assigned to the 
intervention group. All of these factors mean that the resultant allocation was not a simple random one 
(even before attrition). 


In total, 485 individual pupils volunteered to take part in the study, hoping to be part of the summer 
programme. This was much fewer than the developers had planned (1,000), meaning that the trial 
started with considerably less power than intended. The developers were wide-ranging and hard- 
working with their recruitment activities. There were instances where parents and pupils showed initial 
interest in participating in the programme but then dropped out before registration, after registration, 
and before attendance. In Brighton, for example, the first day registration process yielded 45 pupils. Of 
these, five did not then register for the programme, and a further 13 did not turn up for the first day of 
the summer school. The most common reasons were that parents had work or holiday arrangements 
that clashed (6), or pupils were ill (8), changed their minds (2) or did not want to attend as their 
friend(s) had not been selected (2). Only the last of these is due to the embedded evaluation, and so 
the recruitment process must lead to doubts about the feasibility of scaling up the summer school 
programme, and extending it to less populated areas. 


The dropout rate from the treatment group reduced the expected class sizes in all three sites. 


Randomisation 


Prior attainment was estimated using KS2 test and assessment scores for English and maths, and 
these were independent of the randomisation. The randomisation was carried out by the evaluation 
team individually for each pupil (from the 43 schools who provided volunteers), but for the three sites 
(Edmonton, Islington and Brighton) separately. This was originally intended to take place when a 
complete potential participant list was made available at half term of the summer term 2013. However, 
recruitment was delayed, and randomisation took place in three stages (so that families could be 
informed as early as possible). The first stage involved 258 pupils, the second 141 and the third only 
36. In each wave, in each site, 55% of cases were allocated to treatment overall (unbalanced to help 
the developers have viable numbers), and 55% of Year 6 cases were allocated to treatment 
(unbalanced at the request of the funders). This meant that pupils were also randomised separately by 
year group, and so the cell sizes were very small on occasion (such as two or three cases per cell in 
Brighton for the third wave), with almost no variation possible. 


A mechanically shuffled set of cards was used for generating random numbers. There was one card 
for each pupil, with odd/even representing the two groups with the number of each pre-set as above. 
Each pupil was allocated to a group according to the card dealt for them at their turn on the list. The 
list with this group allocation was then sent back to Future Foundations to inform the schools and 
parents for consent, and to register the relevant pupils to attend the summer school programme. One 
pair of twins was treated as one case at the request of the developers meaning that in effect there 
were 434 partly randomised cases. 
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Analysis 


The impact of the trial is represented by the effect size (Hedges’ g) for the post-test raw score results 
of the GL Assessment Progress in English (PiE) and Progress in Maths (PiM) test, contextualised by 
pupils’ prior KS fine point scores in reading, writing and maths. In this report ‘gain’ scores refer to the 
average difference in scores between the prior attainment in fine point scores and the post-test, and 
the raw scores. Such ‘gain’ scores are necessary here because the groups were unbalanced at the 
outset (largely due to dropout after individual randomisation). It would have been preferable to have 
had pre-test scores from exactly comparable standardised tests to the post-tests. However, the GL 
Progress tests selected by the developer do not have comparable alternative versions, and the 
funders wished to reduce the cost of what had been planned as a 1,000 case trial anyway. The pre- 
test scores were rescaled before further analysis to match the range of the post-test raw scores. The 
analysis used was ‘intention to treat’, meaning that all pupils originally identified as eligible were tested 
and their outcomes analysed, regardless of the time actually spent on the intervention. However, a 
high proportion of cases were missing scores. Further analyses include using the same ‘effect’ sizes 
for sub-groups such as FSM-eligible pupils, boys only, and for each year separately. 


Two multivariate regression models were also created. One model used the English outcome score 
(PiE) as the ‘dependent variable, and the other was based on maths (PiM). The predictors were prior 
attainment scores, and pupil background variables all entered in Step 1, and the treatment group 
entered in Step 2. 


Process evaluation methods 


The process evaluation was conducted by the independent evaluators in cooperation with Future 
Foundations. The latter conducted the training of staff, monitored the intervention and collected formal 
records and the views of staff. The evaluators observed the training, the teaching and testing, used 
the texts and documents relating to the intervention, and conducted face-to-face interviews with staff, 
pupils, parents and project members. These interviews were usually conducted without a formal pre- 
set schedule, and arose as the situation allowed. The observations of staff training and 
implementation of the programme in action were as simple and integrated and non-intrusive as 
possible. The schedule of visits was agreed with the Future Foundations team and some interviews 
were arranged at that time. Individuals agreed to be part of the evaluation when agreeing to be part of 
the intervention. 


Attention was given to the training of staff, teachers, mentors and peer-mentors, their receptivity to the 
programme and delivery of the intervention at the three different sites. Two evaluators attended the 
first day of the training when academic content was delivered to the selected teachers. The evaluators 
were participant observers, as which they talked to teachers, mentors and peer-mentors, teacher 
trainers and project leaders to get their perceptions of the programme and the process of curriculum 
development teachers’ recruitment and teachers’ selection criteria. 


The teachers selected for the summer school were all well qualified and experienced in teaching at 
primary or secondary levels. The teacher selection criteria were decided by the Future Foundations 
team aiming to have an effective model of teaching for the four-week programme. The subsequent 
fieldwork in schools included observations of these trained staff delivering the intervention, noting 
inconsistencies or any departures from the programme protocol. The evaluators also considered the 
resources used (such as portfolios, reading materials, feedback sheets), and asked for staff, mentors’ 
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and peer-mentors’ and pupils’ perceptions of these materials. The interviews and field notes were part 
transcribed and shared between the evaluation team. Any school and pupil names have been altered 
to pseudonyms. 


The process evaluation was useful in assessing fidelity to treatment. The perceptions of participants 


provided indications of any resentment or resistance to the programme. They are also useful in 
identifying potential issues or barriers which could be addressed for any future scaling up. 
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Impact evaluation 


Participants 


The school recruitment process was conducted by Future Foundations. The networking process for 
the recruitment involved meetings with the head teachers, and/or senior management team and 
school literacy/numeracy coordinators, and 43 schools agreed to find pupils to take part (their 
summary characteristics appear in the Appendix). 


School participation was not a major issue in the recruitment process. The schools were keen to 
identify pupils who could benefit from the intervention, and were supportive in sending the information 
to parents regarding the free summer school programme. However, the summer holiday was a period 
when engaging pupils to extra schooling was a challenge. Recruitment of individual pupil volunteers 
was lower than expected, and a high proportion of those recruited did not then attend the summer 
programme (Figure 1). According to Future Foundations the main reasons given by the parents were 
their extra engagements, disappointment at friends not attending (due to the randomisation), holiday 
plans and ill health. This level of dropout after randomisation means that the results can no longer be 
considered those of a true randomised controlled trial. 
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Target (n=1,000) 


Excluded (n=0) 


Not meeting 


inclusion criteria 
(n=0) 


; Declined to 
Randomised (n=435) participate (n=0) 


Other reasons (n=0) 


Allocated to intervention (n=239) Allocated to control (n=196) 


Received allocated intervention (n=188) 


Pre-test score provided (n=156) 


Did not receive allocated intervention 
(n=193) 


Provided any pupil data (n=161) 


Post-test score provided (n=164) oa score provided 
n= 


Analysed (n=164) Analysed (n=139) 


Note: some additional pupils provided Note: some additional pupils 
post-test data only provided post-test data only 
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Pupil characteristics 


The pupil characteristics in the achieved sample are provided in Table 1. In order to keep pupil 
selection and group allocation procedures unbiased the following characteristics were obtained only 
after the randomisation procedure. Future Foundations was successful in targeting pupils with various 
possible indicators of disadvantage, and these characteristics were proportionately very similar for 
both achieved groups. 


Table 1. Pupil numbers and characteristics for the trial 


Original Year 5 132 106 
Boys 100 94 
FSM 84 78 
EAL 69 58 
SEN 60 52 
Ethnic minority 133 119 
TOTAL 193 161 


Outcomes and analysis 


The two groups (in terms of the achieved data) were slightly unbalanced from the outset in regard to 
attainment. The treatment group is ahead in terms of KS assessment scores in reading, writing and 
maths (Tables 2 to 4 — note: all tables present data for Years 5 and 6 combined unless otherwise 
specified). This means that a straightforward post-test only analysis would be misleading. The 
question is therefore whether the treatment group has improved in terms of this relative advantage at 
the outset. 


Table 2. KS2 Reading score, pre-test, all achieved scores 


Treatment 169 23.66 4.86 +0.02 
Control 144 23.42 5.76 - 
Overall 313 23.55 5.29 - 


Note: several pupils with post-test scores had no prior data provided by schools. 


Education Endowment Foundation 19 


Impact evaluation 


Table 3. KS2 Writing score, pre-test, all achieved scores 


Treatment 169 22.57 5.03 +0.11 
Control 144 21.94 5.97 - 
Overall 313 22.28 5.48 - 


Table 4. KS2 maths score, pre-test, all achieved scores 


Treatment 169 23.32 6.40 +0.06 
Control 144 22.89 8.12 . 
Overall 313 23.12 7.23 > 


The treatment group also has higher post-test scores on both outcomes (Tables 5 and 6). In English 
the relative advantage for the treatment group is higher than in the pre-test scores, suggesting a small 
gain of around +0.06 (+0.17-0.11) to 0.15 (+0.17-0.02). In maths the relative advantage for the 
treatment group is lower than in the pre-test scores, suggesting a small loss of around -0.02 (+0.03- 
0.05). 


Table 5. PiE score, post-test, all achieved scores 


Treatment 191 39.06 16.08 +0.17 
Control 160 36.36 15.41 - 
Overall 351 37.83 15.82 


Note: several pupils with post-test scores had no prior data provided by schools. 


Table 6. PiM score, post-test, all achieved scores 


Treatment 187 20.87 10.07 +0.05 
Control 159 20.39 11.02 - 
Overall 346 20.65 10.50 


The same result is found when the calculations ignore any cases with any missing pre-test scores, 
and the overall N drops to 313, as in Tables 2 to 4. 
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This result is confirmed when treating the rescaled KS2 fine point scores as if they were true pre-test 
data, and calculating ‘gain’ scores as the difference between PiE or PiM and the pre-test. The pre-test 
score for English is calculated as the mean of the scores for reading and writing. When the ‘gain’ 
scores from rescaled KS fine point scores to PiE or PiM raw scores are considered, there is again a 
positive result for English and a very small negative one for maths (Tables 7 and 8). It is noticeable 
that, insofar as the scores mean anything comparable, the gain in maths over the summer was 
considerably greater than in English for both treatment and control. It is possible either that maths is 
consolidated over summer more than English, even without practice, or otherwise more resilient to 
summer learning loss. English skills, on the other hand, may need the practice that the summer school 
is one way of providing. 


Table 7. Gain score English, all achieved scores 


Treatment 167 3.71 13.24 +0.17 
Control 143 1.46 12.79 : 
Overall 310 2.67 13.06 - 


Table 8. Gain score maths, all achieved scores 


Treatment 164 7.51 8.43 -0.00 
Control 142 7.53 9.24 - 
Overall 306 7.52 8.80 - 


The Year 5 groups were unbalanced at the outset with the treatment group having higher pre-test 
scores than the control in all three subjects. Given the scale of post-allocation dropout, it is possible 
that those pupils who actually attended the summer school were more motivated and perhaps higher 
attaining than those who did not. The treatment group also had higher post-test scores, so the 
question is whether the gain scores show more or less progress. As with the overall picture, there is 
an apparent positive impact on English, but there is also an equivalent positive difference for maths as 
well (Tables 9 and 10). 


Table 9. Gain score English, all achieved Year 5 scores 


Treatment 123 5.24 13.50 +0.12 
Control 103 3.68 12.00 - 
Overall 226 4.53 7.53 - 
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Table 10. Gain score maths, all achieved Year 5 scores 


Impact evaluation 


Treatment 123 7.97 8.38 +0.12 
Control 102 7.01 8.07 - 
Overall 225 7.53 8.24 - 


The Year 6 groups were also unbalanced at the outset, but with the treatment group having lower KS 
point scores than the control in all three assessments. They ended up with lower scores in the post- 
test, so the question is whether the gain scores show more or less progress. Note that because the 
post-test was conducted after Year 6, pupils had transferred to secondary schools, which made testing 
harder and some schools refused to co-operate. Fewer complete scores are available proportionately 
than for Year 5. As with the overall picture, there is an apparent positive impact for English, but there 
is also a large apparent negative impact for maths (Tables 11 and 12). 


Table 11. Gain score English, all achieved Year 6 scores 


Treatment 43 -0.35 11.65 +0.31 
Control 40 -4.24 13.13 - 
Overall 83 -2.23 12.46 - 


Table 12. Gain score maths, all achieved Year 6 scores 


Treatment 40 6.38 8.53 -0.25 
Control 40 8.87 11.71 - 
Overall 80 7.62 10.25 - 


Considering only the achieved sample for FSM-eligible pupils, the results are similar to the overall 
results. FSM-eligible pupils in the intervention appeared to make progress in English but to have gone 
backwards in maths compared to the control group (Tables 13 and 14). 


Table 13. Gain score English, all achieved scores, FSM-eligible pupils only 


Treatment 78 3.68 13.02 +0.16 
Control 73 1.58 12.78 - 
Overall 151 2.67 12.90 - 


Education Endowment Foundation 


Impact evaluation 


Table 14. Gain score Maths, all achieved scores, FSM-eligible pupils only 


Treatment 77 6.31 8.57 -0.12 
Control 73 7.28 8.27 - 
Overall 150 6.78 8.41 Fi 


The same kind of difference between English and maths also applied to the results considered for 
boys only (Tables 15 and 16). If the intervention is deemed to have had a benefit for progress in 
English then this is also true for FSM-eligible pupils and boys. There does not seem to be any benefit 
in maths. 


Table 15. Gain score English, all achieved scores, boys only 


Treatment 87 3.03 13.24 +0.22 
Control 84 0.16 13.06 - 
Overall 171 1.62 13.19 - 


Table 16. Gain score maths, all achieved scores, boys only 


Treatment 83 7.08 8.73 +0.01 
Control 84 6.95 9.97 - 
Overall 167 7.01 9.37 - 


Table 17 presents the R values for two regression models, each based on two steps. The first model 
is used to try and explain variation in the post-test PiE score; the second concerns the post-test PiM 
score. In Step 1, the pupil background and prior attainment scores for the relevant subject(s) are 
included, and Step 2 adds the binary variable for being in the treatment group or control. The model is 
better at explaining variation in the post-test outcome for English than maths, perhaps partly because 
there are two estimates of prior attainment (reading and writing KS scores). But for both models the 
bulk of the variation that is ‘explained’ by the variables in the model is explained at Step 1. Once pupil 
background and prior attainment is accounted for, very little difference is made by knowing whether a 
pupil was in the treatment group or not. Knowledge of the treatment in this simple contextualised (by 
pupil background) model adds only very slightly to the accuracy of the prediction. This model is not, in 
itself, any test of causation but it does provide a caution about the strength and importance of the 
intervention in relation to pupil prior characteristics. 
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Table 17. Variation explained (adjusted R-squared) in two-stage regression model, using two 
possible outcomes 


Step 1. background and | 0.35 0.42 
prior attainment 


Step 2. treatment group § 0.36 0.42 


For completeness, Table 18 presents the coefficients for all variables retained in either model. The 
largest of these by some way is the pre-test score in the relevant subject(s). Being in the treatment 
group is then slightly positively related to the outcome for English (standardised coefficient +0.08), and 
slightly negatively related for maths (standardised coefficient -0.01). To that extent, the models simply 
repeat the headline findings. 


Table 18. Gain score English and maths, all achieved scores, boys only 


Step 1 

FSM -+0.00 -0.02 
Sex (female) +0.05 +0.01 
SEN -0.05 -0.05 
EAL -0.02 -0.09 
Ethnicity (White UK) +0.07 +0.08 
Prior attainment +0.42 +0.39 
(reading) 

Prior attainment +0.38 +010 

(writing) 

Prior attainment (maths) -0.23 +0.24 
Step 2 

Treatment (or not) +0.08 -0.01 


It is important to recall that the evaluation was designed as a straightforward comparison of the 
outcomes between two groups for all pupils. Modelling and the analysis of sub-groups can yield 
interesting insights but do not themselves have the power of a trial. Nor can they overcome problems 
such as participant attrition. 
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Dosage 


The evaluation was to have included an analysis of dosage, but this was not possible due to the 
incomplete attendance figures provided. 


Cost 


The summer school was a complete four-week programme including teaching and teaching materials, 
enrichment activities, breakfast, lunch, and some transport. 54 teachers were trained, and 34 were 
then hired. The total cost, including evaluation, was between £500,000 and £550,000 for 193 pupils 
(not all of whom participated in this evaluation) across three sites. The achieved pupil to teacher ratio 
was around 5.5, and each class also had a peer-mentor and a mentor available. 


Excluding evaluation costs, the cost of the approach is estimated at £1,370 per pupil. This estimate 
includes administration, resources and activities (estimated at £350 per pupil), salary costs and 
training (£835) and food and transport (£185). Estimates are based on 160 pupils attending a school 
on a single site. 
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Process evaluation 


Implementation 


Future Foundations improved important aspects of the summer school programme in view of its prior 
experience and the recommendations given in the formative evaluation report of 2012. The academic 
content for literacy and numeracy classes was improved in terms of practice materials. The syllabus 
increased the range of exercises where pupils could revise concepts they found difficult. There were 
also various opportunities for pupils to give feedback to teachers on their teaching. 


The staff training was conducted for all teachers, mentors and peer-mentors who were planned to be 
employed at three different sites. In the event, not all could be employed due to low recruitment of 
pupils. The training was well conducted and structured, with many examples of activities teachers 
could use. Teachers were generally motivated and eager to try something new. They seemed positive 
and to enjoy the activities. The role of mentors was defined in terms of academic support to pupils in 
the classroom rather than just assisting teachers in executing the lessons. During the visits, mentors 
were seen supporting pupils in the classrooms, and specifically those who regularly needed the 
individualised attention of an adult. 


A primary school model was followed for the settings, in which pupils do not have to change 
classrooms. Teachers came to the classrooms and all teaching materials were made available to the 
pupils there. This model was adopted with a view that primary school pupils are more used to this 
format of classes and there would be more chance to emphasise teaching and learning activities 
rather than making them adjust to the new routines. On the other hand, this reduced the chances for 
experiencing what would happen in secondary school. 


The planned literacy and numeracy sessions were closely followed on all sites. The implementation of 
literacy sessions was very close to the developed lesson plans. However, numeracy sessions were 
observed to change with time into individualised tutoring sessions. One of the reasons was perhaps a 
wide range of abilities in some class groups, and teachers, with the help of mentors, broke numeracy 
classes into further small groups or one to one sessions. 


The number of pupils in each classroom was less than expected. There were usually three or even 
four adults in each classroom of 10+ pupils, and every child was seen receiving extra attention and 
time from the adults. In some observations there were more adults than pupils in each classroom. This 
was partly due to low recruitment, and partly due to pupil absences. One class observed was at half 
strength. The results of the current programme need to be interpreted in the light of these high ratios. 


The intervention project team had a system to monitor and give feedback to teachers on all three 
sites. Each school had been assigned a coordinator who looked at progress on the syllabus followed 
in the teaching, and gave regular feedback to teachers on the quality of teaching. In one case it was 
reported that a teacher was asked to leave the programme following multiple unsatisfactory 
observations. 


Several lessons observed by the evaluators were poorly taught especially for maths. Basic 
pedagogical and factual errors were observed, and in one case pupil written responses were marked 
incorrectly. In literacy especially, more sessions were seen to be fun and enjoyable for all. Despite the 
pupil ratios, class control was sometimes poor. 


Barriers to delivery 


All teaching staff were committed to two weeks of the summer school programme. In the third week 
there was a change of all teaching staff which was a major transition to adjust to in a short programme 
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of just four weeks. In order to overcome the effect of this transition, mentors and peer-mentors 
remained the same. The mentors kept regular monitoring details of each class they were supporting. 
This monitoring record was given to the new teachers in the third week in order to help them 
understand individual pupils’ academic performance and behaviour in the class. This seemed to work 
better than it had in the pilot. 


The classes were combined year groups. The older pupils were sometimes more aware of the 
concepts being taught in the class and therefore the given tasks were easier for them to complete. It 
was also observed that Year 6 pupils were doing some activities for the Year 5 pupils, and so not 
letting them learn independently. The groups were not homogeneous in terms of abilities and levels. 
Some pupils were not necessarily in need of a literacy and numeracy catch-up intervention, and the 
taught content was not up to the level of all pupils, especially those who were in Year 6. This variety 
was sometimes a challenge for teachers and perhaps the reason that teachers were observed losing 
control (even in classes with fewer than two pupils per adult). In some of the observed lessons, more 
time was given to class management instructions than the teaching task or discussions on the topic. 


Future Foundations selected teachers according to the criteria of teaching experience, enthusiasm for 
teaching and willingness to support disadvantaged pupils. This resulted in considerably more 
experienced teachers than the ones selected in the 2012 programme. However, even so the standard 
of teaching was mixed. 


Is the intervention attractive to stakeholders? 


This summer school programme was created as a literacy and maths catch-up intervention during the 
summer holidays. The curriculum and syllabus is not much different from what is followed in ordinary 
schools, which some of the pupils most in need of catch-up may not enjoy (although this was not a 
view expressed particularly strongly by the pupils themselves). Extra activities were integrated which 
supported the learning. Overall, the atmosphere in each site was good. 


Interviews with parents suggested that the free summer programme kept their children engaged in 
reading and maths. Parents reported their satisfaction concerning the quality and overall management 
of the programme. Some of them also stated that they could not afford expensive holiday trips for their 
children so such an arrangement provided opportunities to their children which they could not have 
managed otherwise. 


Pupils were generally enthusiastic about attending the summer schools. Some of them also reported 
that it was the afternoon activities they enjoyed more than the teaching sessions. They said that the 
teachers and mentors were very supportive and several pupils wanted to come back to the 
programme in the next summer. Several were returners who had been in the pilot programme in 2012. 


Outcomes 


Perceived outcomes 


Teachers in the summer schools reported that the syllabus given to them was very well structured. 
They did not need to prepare more than was given in the detailed lesson plans, although some further 
planning and adaptation was possible. The objectives, timings, use of materials and feedback, 
according to teachers, were all fully integrated in the plans. However, the teachers reported that this 
was not always appropriate when dealing with classes of mixed year groups and abilities. Some 
flexibility was needed. The teachers for numeracy were also observed to have divided the numeracy 
classes into smaller groups of two to three pupils as they found that not all the pupils could follow 
similar plans. In one numeracy session pupils with SEN were working with mentors all of the time and 
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they were given materials and practice sheets which were different from the syllabus planned for the 
whole class. This was less an issue for literacy activities. 


The mentors working with pupils with special educational needs reported that regular tutoring support 
had increased the confidence of pupils and in the fourth week of the programme they felt that the 
pupils had improved. Pupils on all three sites were observed to have received support in literacy and 
numeracy almost at the level of private tutoring. The teachers suggested that individualised attention 
had increased the confidence of pupils. 


Regular attendance of summer school was also perceived to be an indicator of success by parents, 
teachers and mentors. Most pupils, on all three sites, were reportedly attending the school regularly, 
and they had enjoyed coming despite it being during the holidays. Because the programme was ina 
secondary school, pupils could gain a different type of school-experience, and a different approach to 
teaching and learning. This could help them with the transition to secondary school. Also, through 
being out of their usual roles as possible ‘naughty’ or ‘silly’ kids they could have a fresh start with all 
new teaching and management staff members. Mixing the year groups 5 and 6 was perceived by 
some pupils as a fun experience and a positive influence on their learning. 


Fidelity 


The intervention was followed and maintained to similar standards on all three sites by Future 
Foundations. On each site there were managerial staff appointed to maintain the daily routines 
according to the given plans. Academic aspects were regularly monitored by the coordinators. The 
pupils in the treatment group have experienced a similar number of teaching hours for literacy and 
numeracy sessions. 


The recruitment of the teaching staff and mentors was conducted by the same panel members of the 
Future Foundations team. There was no clear difference between recruited staff members on all three 
sites. 


The most adaptable segment of the programme was extra academic and outdoor activities. The 
afternoon activities sometimes differed because the providers of activities were different on the three 
sites. The range of possible providers and local hosts was greater in London than in Brighton. 


Formative findings 


The programme was intended to support struggling pupils to catch-up in literacy and numeracy 
learning. The pupil selection and recruitment procedure was mainly dependent on volunteering pupils 
and parents, and recruitment was difficult. Whilst recruitment materials and messaging for schools 
made clear the pupil selection criteria, limited access to pupil demographic data early enough before 
the intervention meant it was impossible to verify all applications. Therefore the final selection had a 
higher proportion than desirable of pupils who were above the intended levels of the programme. This 
needs to be prevented in any future activities. 


Similarly, in most cases, teachers did not have the data required to divide pupils into class groupings 
following a consideration of their needs for literacy and/or numeracy. In order to manage poor 
behaviour and discipline some groupings were changed, but apparently no such decision was taken 
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on the basis of pupils’ performance. The short span of four weeks is very limited to identify and make 
changes, but a prior consideration of the needs of the selected pupils might have improved the 
effectiveness of the targeted programme. 


The evaluators observed that pupils with behavioural issues were taken out of the sessions for 
mentoring. They met with a behaviour mentor to discuss their behaviour and followed a process for 
being reintroduced to the class. This was introduced in the 2012 pilot and specifically trained staff 
from secondary schools were recruited and selected for this role. This was done both to help prepare 
that pupil for returning to class, and to maintain the flow and discipline of the classes, so avoiding the 
loss of learning for other pupils. However, there was no extra academic hour or activity to support that 
child’s loss in learning. 


The post-tests used were the pencil-and-paper versions of PiE and PiM. GL Assessment developed 
these standardised tests which were posted to each of the addresses of the schools attended by 
pupils who had participated in the programme. Future Foundations worked closely with the evaluators 
and GL Assessment to get the post-tests completed in standard conditions. The evaluation team 
members visited some schools to observe the testing process, which was found to be good. Some of 
the SEN pupils needed staff support to read the questions to them and this was observed to be done 
in a very neutral way. Most of the secondary schools now attended by the previous Year 6 pupils were 
not previously part of the intervention, and some were less co-operative with the post-test than the 
primary schools. This led to higher attrition of scores for Year 6 than Year 5. 


Control group activity 


The control group carried on with summer holidays, and so nothing specific is known about their 
activities. 
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Conclusions and implications 


Limitations of study 


The low recruitment, and high dropout of pupils after random allocation to the groups, has reduced the 
sample size and so any ability to make generalisations or draw conclusions. Due to this low 
recruitment reducing the power of this trial and the high dropout after allocation introducing a high 
probability of bias, this is not the definitive result that was planned. It is still not clear whether summer 
schools work as intended to help struggling pupils to catch-up. 


Interpretation 


This was a well-run intervention, generally supported by teaching staff and pupils. To achieve this 
level of co-ordination and support required considerable energy and monitoring. The class sizes were 
very small, making provision even more expensive than planned. If the primary purpose is literacy and 
maths catch-up there are already more effective and cheaper alternatives. Indeed, the evidence is that 
this programme either makes no difference to attainment in maths or actually damages it. This could 
have been due to a difference in the quality of the teaching observed, or it could be that maths is less 
susceptible than literacy to summer learning loss, but these remain speculations. There is more 
promise for attainment in literacy, but it is no more than a strong promise until demonstrated more 
convincingly by a randomised controlled trial or similar (perhaps using a cross-control design that 
commits all volunteers to some summer activity whether summer school or an unrelated alternative). 
The probable gain in literacy is similar to the number of weeks of actual literacy teaching. 


Future research and publications 


Three important questions remain unanswered. Why is there such an apparent difference in the 
results for English and maths? Can summer programmes really improve average attainment in 
literacy, with the promise that this holds for access to the rest of the secondary school curriculum? 
And does such a summer programme have wider benefits, such as in terms of confidence and 
enjoyment not directly measured in this evaluation? 


In the near future, the evaluators plan to prepare a summary paper describing the evaluation for a 
peer-reviewed journal. 
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Appendix 


yNo) ol=Valolb.4 


These are the school-level characteristics of the schools that provided pupils or test results for the 
trial. 


Academy 11 to 

Secondary Sponsor led 19 London 938 42.9 70 11.4 24.1 27 
Academy 11 to 

Secondary Sponsor led 19 London 1408 47 64.3 7.2 24.2 63 
Comprehensive 11 to 

Secondary School 19 Brighton 624 39.7 8.3 15.2 24.6 61 
Community 

Primary School 3to11 London 238 56.3 71.1 17.2 25.2 
Academy 11 to 

Secondary Sponsor Led 18 London 759 52.7 51.3 15.8 25.4 
Voluntary 

Primary Aided School 3to11 London 381 29.3 49.8 7.3 26 25 
Community 

Primary School 3to11 Brighton 308 48 9.8 21.8 26.1 24 
Community 

Primary School 7 to 11. London 585 40.2 78.3 7.5 26.2 22 
Community 

Primary School 3to11 London 256 61.6 54.7 13.3 26.2 
Community 

Primary School 3to11 Brighton 350 33.6 7.8 12 26.4 19 
Voluntary 

Primary Aided School 4to11 London 202 52 52.2 18.8 26.4 10 
Comprehensive 11 to 

Secondary School 19 Brighton 783 19 4.9 16 26.6 33 
Academy - 
Converter 

Primary Mainstream 3to11 London 870 28 54.4 12.1 26.8 13 
Community 

Primary School 7 to 11. London 582 50.3 70.8 5 26.9 13 


PN oy ol-Valo lpg 


Voluntary 

Primary Aided School 3to11 London 683 33.4 47.7 7.6 27.2 19 
Community 

Primary School 3to11 London 479 45.5 73.1 5.8 27.4 3 
Community 

Primary School 3to11 Brighton 422 14 3.6 13.5 27.4 27 
Community 

Primary School 5to11 London 460 32.8 43.6 8.3 27.6 11 
Community 

Primary School 3to11 Brighton 402 42 6.6 19.9 27.8 7 
Community 

Primary School 3to11 London 210 51.3 62.5 8.6 27.8 6 
Community 

Primary School 3to11 London 715 34.4 52 10.3 27.9 10 
Community 

Primary School 3to11 London 657 40.6 61.4 5.2 28 3 
Community 

Primary School 3to11 Brighton 382 36.4 18.6 15.2 28 17 
Community 

Primary School 3to11 London 375 59.8 32.8 17.1 28.1 5 
Community 

Primary School 3to11 London 247 48.2 68.5 7.7 28.2 
Community 

Primary School 3to11 London 645 50 74.6 6.4 28.3 3 
Community 

Primary School 3to11 London 327 57.6 72.3 14.1 28.3 
Voluntary 

Primary Aided School 4to11 London 179 63 20.8 11.7 28.4 
Community 

Primary School 3to11 London 939 42.2 66.6 9.6 28.4 3 
Voluntary 

Primary Aided School 3to11 London 401 57.3 55.9 8.7 28.6 4 


PN oy ol-Valo lpg 


Community 

Primary School 3to11 London 467 39.8 27.4 5.6 28.6 5 
Community 

Primary School 3to11 London 481 47.3 73.1 10.4 28.7 8 
Community 

Primary School 3to11 London 363 58 59.9 9.6 28.7 9 
Voluntary 

Primary Aided School 4to11 London 435 23.3 49.9 6.2 28.8 14 
Voluntary 

Primary Aided School 4to11 London 197 51.8 32.9 18.3 28.8 14 
Community 

Primary School 3to11 London 463 44.9 69.6 9.5 29.2 3 
Community 

Primary School 3to11 London 341 50.1 67.3 6.2 29.4 7 
Community 

Primary School 3to11 London 232 37.2 50.9 4.7 29.7 
Community 

Primary School 3to11 London 781 38.8 78.4 9.7 29.8 2 
Community 

Primary School 3to11 London 336 40.4 24.6 9.2 29.9 5 
Voluntary 

Primary Aided School 4to11 London 198 45.5 24.9 3.5 30.1 0 
Academy - 
Converter 

Primary Mainstream 3to11 London 442 27.4 24.4 11.8 30.7 0 
Community 

Primary School 3to11 London 462 41.2 53.2 13.2 31 3 


You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 or 
email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission from 
the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk 


A Education 


Endowment 
Foundation 


The Education Endowment Foundation 

9th Floor, Millbank Tower 

21-24 Millbank 

London 

SW1P 4QP 
www.educationendowmentfoundation.org.uk 


